Fundamentals of Missing Data in Evaluation

Presentation to MSU Department of Psychology, Program Evaluation Occasional Speaker Series, East Lansing, MI

Steven J. Pierce

pierces1@msu.edu

Center for Statistical Training and Consulting

2024-12-05

Outline

What is missing data?
Why do we end up with missing data?
Why should we care about missing data?
How can we diagnose the missing data issues for a given study?
What should we do about missing data?
- Prevention
- Diagnosis
- Treatment
Advice

What is missing data?

Missing data (MD) are measurements you want or intended to collect but did not get.^[1]

Having MD is common in research & evaluation studies.
If you do much evaluation work, you will run into MD.

Why do we end up with missing data?

Data collection doesn’t always go according to plan…

Human Factors	Other Factors
Participant behavior	Equipment failures
Evaluator errors	Records/Databases
Partner behavior	Unusual Events

Like many things in life, data collection doesn’t always go according to plan. > Human factors often contribute to the occurrence of missing data. For example, a survey participant may skip a question without recording any response, offer an illegible or irrelevant answer, refuse to answer it, or simply say “I don’t know”. They may skip whole instruments or entire data collection visits. But evaluators occasionally make errors that lead to missing data. Imagine forgetting to take a data collection form with you, or programming the skip pattern for an online survey incorrectly. Sometimes, our evaluation partners’ decisions or behavior can prevent obtaining some data we wanted. Perhaps they object to sharing something with you, or were not diligent about logging some events of interest.

Other things can go wrong too. Perhaps a physical instrument breaks, or its battery dies, or your online survey app crashes. If you’re using an organization’s records or databases, those may be incomplete, difficult to search for relevant data, coded in unhelpful ways, or could have been lost or damaged or even destroyed according to some retention policy. In terms of events, consider the recent hurricanes in Florida that disrupted data collection for one of my projects.

Missing Data & Project Lifecycle

Why should we care about missing data?

Ethics for Evaluators

Handling missing data well enacts our guiding principles^[2]:

AEA logo.

Systematic inquiry
Competence
Integrity

Perhaps the broadest answer to why we should care about missing data is that this is an ethics issue. Thoughtful handling of missing data is a good way to enact several of AEA’s guiding principles for evaluators.

> Systematic inquiry demands that we examine the quality of the available data and use it appropriately. We have an obligation to use sound methodology. There are scientific consequences of missing data but there are also legitimate, evidence-based methods for minimizing its impact on our analyses, conclusions, and products.
> Competence demands that we invest in skills necessary to understand and handle missing data properly because doing it poorly may compromise our evaluations. Our funders, clients, and stakeholders–including the program staff and the people and communities they serve and operate within–need and deserve competent evaluation services.
> Integrity demands that we be transparent about the amount of missing data, how we handled it, and how it may affect our evaluation results.

Scientific Activities^[3]

There are 3 major scientific activities that can be affected by missing data.

Making structured observations of constructs.
Using observations to draw inferences about relationships between constructs.
Generalizing the results to populations beyond the collected sample.

Consequences for Measurement^[3]

Availability of constructs
Decreased reliability due to increased error variance
Bias from poor content coverage
Construct validity

Missing data can affect the availability of a construct for your analyses. The extreme case would be a construct measured by a single variable. If that’s missing, then the entire construct is missing for that individual and it may be very difficult to recover that information. Missing data is still a problem when we use multiple indicators to measure a construct, though perhaps less severe, and it may depend on how you combine items to create the score.

Missing data decreases the reliability and construct validity of our measures. Internal consistency reliability decreases because missing data increases the error variance in a measure. You may recall that reliability is a function of how many items are used, so fewer observed items leads to lower reliability.

Further, some items may be more important indicators of a construct than others. You may get a biased estimate of the construct due to poorer content coverage if they are missing. Construct validity depends on both reliability and content coverage, so missing data is a threat to validity.

Consequences for Internal Validity^[1,3]

Selection bias
Compromised randomization
Power and precision
Inaccurate model assumptions

Missing data is a threat to a study’s internal validity for several reasons. Selection bias can result from systematic differences between subgroups of participants on constructs relevant to the study conclusions. For example, those subgroups could be treatment versus control groups, people who complete the study versus drop out, or those who have complete data versus incomplete data. These differences may manifest with respect to demographic variables, covariates, focal predictors, mediators, moderators, or outcomes. Imagine evaluating a longitudinal health study where people who start out with worst symptoms drop out before the posttest more often than those with mild symptoms. Dropping out leads to missing data at posttest. If we fail to handle that properly in the analysis, we may draw inaccurate conclusions about the longitudinal trend in outcome scores. Or perhaps the retention rates differ between treatment and control groups in that same study. Selection bias from such differences reduce the sample representativeness and become rival explanations for findings, making them a threat to internal validity.

We use randomization to make groups equivalent on measured and unmeasured confounding variables as a way to strengthen validity. That works best in large samples. However, missing data can compromise that effect of randomization by reducing sample size. It can also reduce randomization’s ability to achieve equivalence between groups on confounding variables when data are missing differentially across the groups. That reduces internal validity.

Most statistical software defaults to listwise deletion of cases that have any missing values on the variables involved in an analysis. Reducing the sample size will always reduce both power to detect effects and the precision of effect size estimates. In addition to reducing overall sample size, missing data may lead to more unbalanced sample sizes between groups in a study, which also reduces statistical power in many models.

Finally, missing data can change the distributions of variables in ways that invalidate the assumptions of various statistical models. Some models are more robust than others, but making inaccurate modeling assumptions risks running analyses that will yield misleading conclusions. That’s another threat to internal validity.

Consequences for Generalizability^[1,3]

A representative samples is crucial to generalizing to the intended population!

Theory development & cumulative knowledge
Policy & decision-making

We draw samples to study populations when it isn’t feasible to collect the data we want from the entire population. The key to getting inferences that properly generalize from a sample to the intended population is getting a representative sample. Biases creep into our findings when we our samples aren’t representative. Missing data can introduce systematic biases into our samples. Suppose you’re studying a behavior that’s illegal or stigmatized in some way. Some people who engage in it may be relucant to provide all the data you want. If we ignore the missing data problem, listwise deletion will result in analyzing the subset of the intended population that’s willing to answer our questions. That in effect changes the population to which we could legitimately generalize our findings from “people who engage in behavior x” to “people who engage in behavior x and provided complete data”. Many people fail to attend to that though and assume their results still appply to the intended population.

That has consequences for developing and testing scientific theories and accumulating knowledge across studies. An example is the effect of publication bias, also called the file drawer problem, on meta-analyses. We know that studies with non-significant findings are less likely to get published. That means the published literature has a missing data problem that can seriously distort the results of meta-analyses that aim to summarize results of a collection of published papers.

Finally, the big picture here is that when missing data impair generalizability, we may be using inaccurate or misleading findings to inform public policy and important decisions.

Types of Missingness

Item-level
Construct
Person-period
Planned vs unplanned

Mechanisms of Missingness

Missing completely at random (MCAR)
Missing at random (MAR)
Missing not at random (MNAR)

MCAR

MCAR is when neither observed nor unobserved variables predict which data is missing.

MAR

MAR is when observed values predict which data is missing.

MNAR

MNAR is when unobserved values predict which values are missing.

Diagnosis

Describing the Amount of MD

^[3]

Numbers & percentages of complete & incomplete cases
Number and percentage of missing values for each variable
Nature and frequency of missing data patterns

Predictors of Attrition & Missingness in Longitudinal Studies

Study arm: Compare retention rates
Study site (in multisite studies)
Baseline/pretest values of outcome variables may predict who drops out or has missing values
Other covariates (demographics, site, )

Treatment

Prevention

An ounce of prevention is better than a pound of cure

^[4],^[5]

Advice

Collaborate with a statistician!

Practical Options

Item-level missingness in scale scores^[6,7]

References

1. Fernández-García, M. P., Vallejo-Seco, G., Livácic-Rojas, P., & Tuero-Herrero, E. (2018). The (ir)responsibility of (under)estimating missing data. Frontiers in Psychology, 9(556). https://doi.org/10.3389/fpsyg.2018.00556

2. American Evaluation Association. (2018). Guiding principles for evaluators [Web Page]. Author. https://www.eval.org/About/Guiding-Principles

3. McKnight, P. E., McKnight, K. M., Sidani, S., & Figueredo, A. J. (2007). Missing data: A gentle introduction. Guilford Press.

4. Leeuw, E. D. de. (2001). Reducing missing data in surveys: An overivew of methods. Quality & Quantity, 35(2), 147–160. https://doi.org/10.1023/A:1010395805406

5. Wisniewski, S. R., Leon, A. C., Otto, M. W., & Trivedi, M. H. (2006). Prevention of missing data in clinical research studies. Biological Psychiatry, 59, 997–1000. https://doi.org/10.1016/j.biopsych.2006.01.017

6. Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. https://doi.org/10.1146/annurev.psych.58.110405.085530

7. Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4), 372–411. https://doi.org/10.1177/1094428114548590